This assignment will be your opportunity to pull all the tools and techniques you have learned in this course and apply it to the real-world problem of identifying patients with diabetic complications from clinical text notes. Specifically you will work with a corpus of notes from patients who have diabetes. You will work to identify which notes identify patients who have diabetic complications of neuropathy, nephropathy, and/or retinopathy and which type of complication the note identifies.

In order to accomplish this task you will:

• Decide on and apply one or more text processing techniques and describe the key metrics you used for that technique

• Explain one regular expression used in your project

• Assess your text processing algorithm performance by identifying how many notes were correctly and incorrectly classified.

• Describe why your method may not have been accurate for that note and identify an approach you might use to correctly classify those notes in the future.

You will be expected to submit two documents:

1. A presentation slide deck

2. Your code used to perform the project (ideally as an RMarkdown document)

You will be assessed on :

1. Whether you select a text processing and completely describe the key metrics for your chosen method  (i.e., for note section method - description of note section/s and headers used to identify note section; for keyword window, the initial keyword and window size)

2. Whether you explain how at least one regular expression worked and give examples for what words/phrases it would detect.

3. Whether you provide a count of the number of notes you are able to correctly and incorrectly classify.

4. Whether you provide 1-2 examples of notes that you correctly or incorrectly identifies and describe how your method does or does not identify them.

5. Describe an alternative approach you might use to identify at least one note you incorrectly classified.

Project Data:

You will use a set of clinical notes (synthetic from the www.MTSamples.com website) from patients with diabetes. These notes can be found on Google BigQuery in the learnclinicaldatascience project - course4_data.diabetes_notes

When you are ready to determine how accurate your approach is, you can use the data in course4_data.diabetes_goldstandard

Clinical Background Information (Summarized from the Mayo Clinic ):

What is diabetes?

• Diabetes is a group of diseases that affect how bodies process blood sugar and can result in blood sugars being too high or too low.

What are diabetic complications?

• When diabetes is not well controlled (gets too high or too low regularly), other conditions can develop - these conditions are called "complications". In this assignment we are going to look at three common types of diabetic complications: diabetic neuropathy, diabetic nephropathy, and diabetic retinopathy.

What is diabetic neuropathy?

• Neuropathy is the feeling of tingling, numbness, burning, or pain at nerve endings. Diabetic neuropathy occurs when too much sugar in blood affect the blood vessels that feed nerves. This typically starts in the tips of the toes and fingers, and can spread as the damage increases.

What is diabetic nephropathy?

• Nephropathy is kidney damage that can lead to kidney failure - like neuropathy it is caused by regularly having too much sugar in the blood.

What is diabetic retinopathy?

• Retinopathy describes damage to the retina that can cause blindness. It is caused by damage to the blood vessels of the retina due to too much sugar in the blood.

Project Template:

This is a template you can use for your presentation if you choose: